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Abstract 

We employ the face recognition technology developed in house at 
face.com to a well accepted benchmark and show that without any tuning 
we are able to considerably surpass state of the art results. Much of the 
improvement is concentrated in the high- valued performance point of zero 
false positive matches, where the obtained recall rate almost doubles the 
best reported result to date. We discuss the various components and inno- 
vations of our system that enable this significant performance gap. These 
components include extensive utilization of an accurate 3D reconstructed 
shape model dealing with challenges arising from pose and illumination. 
In addition, discriminative models based on billions of faces are used in 
order to overcome aging and facial expression as well as low light and over- 
exposure. Finally, we identify a challenging set of identification queries 
that might provide useful focus for future research. 



1 Benchmark and results 

The LFW benchmark [B] has become the de-facto standard testbed for uncon- 
strained face recognition with over 100 citations in the face recognition literature 
since its debut 3 years ago. Extensive work [TS1 H31 ISl El El SI HS1 El IH1 El KH HE] 
has been invested in improving the recognition score which has been consider- 
ably increased since the first non-trivial result of 72% accuracy. 

We employ face. corn's r2011b2 face recognition engine to the LFW bench- 
mark without any dataset specific pre-tuning. The obtained mean accuracy is 
91.3% ± 0.3, achieved on the test set (view 2) under the unrestricted LFW 
protocol. Figure [l] (a) presents the ROC curve obtained in comparison to pre- 
vious results. Remarkably, much of the obtained improvement is achieved at 
the conservative performance range, i.e., at low False Acceptance Rates (FAR). 



1 face.com has a public API service pQ which currently employs a previous version of the 
engine. 




Eigenfaces, original 
Nowak, funneled 
Merl 

Merl + Nowak, funneled 
Hybrid descriptor-based, funneled 
Vl-like/MKL, funneled 
Hybrid, aligned 
Combined b/g samples based, aligned 
Attribute and simile classifiers 
LDML+MkMIM, funneled (u) 
Multishot combined, aligned (u) 
LBP multishot, aligned (u) 
Multiple LE + camp 
LBP + CSML, aligned 
CSML + SVM, aligned 
High-Throughput Brain-Inspired Features 
Combined PLDA, aligned & funneled (u) 
Associate-Predict 
face.com r2011b (u) 



0.1 



0.2 



0.3 



0.4 



0.5 



0.6 



false positive rate 



0,7 



(a) 



0.9 




:lassifiers 
' LDML+MkNN, funneled (u) 
Multishot combined, aligned (u) 
LBP multishot, aligned (u) 
Multiple LE + comp 
LBP + CSML, aligned 
CSML + SVM, aligned 
High-Throughput Brain-Inspired Fe atures 
r^mhin"d PI n A , 

Associate-Predict 
face.com r2011b (u) 



0.004 0.006 
false positive rate 



(b) 



Figure 1: ROC curves for View 2 of the LFW data set. Each point on the curve 
represents the average over the 10 folds of (false positive rate, true positive rate) 
for a fixed threshold, (a) Full ROC curve, (b) A zoom-in onto the low false 
positive region. The proposed method is compared to scores currently reported 
in http:/ /vis-www. cs. umass.edu/lfw/results. html 

Specifically, for FAR=0 the recall (TPR) is over 55%, which is significantly 
higher than all previously reported results, as shown on Figure [l] (b). 

As can be seen in Figure [6j the false matches arise in circumstances that are 
considerably difficult even for humans to recognize. This is often the result of 
extreme personal makeovers (much of LFW is concerned with celebrities) and 
challenging imaging conditions. Anecdotally, using the obtained results, the 
system was able to identify a newly discovered error among the thousands of 
labels of the benchmark when it discriminated clearly between the two basketball 
coaches named Jim O'Brien. 
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Figure 2: Pose correction (middle) to a non-frontal input 70x70 image (right) 
with arbitrary lightning (left) 




Figure 3: Input image (left most) and its 3D reconstructed model shown from 
different angels, with/without texture and anthropometric points 

2 Methods 

Face.com has been used by users and developers to index almost 31 billion 
face images of over 100,000,000 individuals. Leveraging this immense volume 
of data presents both a unique opportunity and an unusual challenge. The 
capability developed in house in order to make use of this data builds upon 
various achievements in scientific computation, database management and ma- 
chine learning techniques. The run-time engine itself is a real-time one, able to 
process face detection together with recognition of over 30 frames per second 
on a single Intel 8-core server machine based on the Sandy Bridge architecture 

One key direction in which the large volume of data is utilized is in the 
development of a proprietary 3D face reconstruction engine. This engine is 
able to produce an accurate 3D model from a single unconstrained face image. 
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Figure 4: Another 3D reconstruction sample. Input image (left), Shape only 
rendering (right) with an arbitrary view rendering (middle) 




Figure 5: Duane Lee Chapman 0001 from the LFW dataset with arbitrary 
lightning imposed on its reconstructed 3D model 

Unlike 3DMMs [2J, face. corn's 3D system works in real-time and is robust enough 
to handle general unconstrained imaging conditions in rather low-resolution 
images, see Figures [5J [3j and [3] for examples. 

Once 3D reconstruction is obtained, two of the biggest challenges in face 
recognition become well defined and tractable. Namely, the face recognition 
engine is able to largely overcome pose and illumination variations. Pose is dealt 
by a normalization process in which all images are mapped to a frontal view. 
Unlike previous works [111 [13] [^] that tried to achieve view normalization without 
3D modeling, outer plane rotation is accurately handled. The 3D model also 
enables the re-illuminating or rather delighting of the model once the parameters 
of the light sources are estimated, see Figure [5] 

Some variations in face images of the same individual arise from aging or 
expression and are hard to model directly. By employing non-parametric dis- 
criminative models trained with tens of millions of data pieces, we are able to 

2 various other contributions have also employed LFW-a which is an aligned version of LFW 
obtained using the face.com API j as well. 
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extract learned complex features that are invariant to these factors. Specifically, 
these features are based on building blocks that are selected exemplars from our 
repository, which are used to classify new probes as well as estimating attributes 
such as ethnicity, age and more. 

Despite considerable improvement over state of the art results, performance 
is still not perfect, and some image pairs are mislabeled, see Figure [6] In 
order to promote the research of difficult cases, we are releasing] the full list 
of view 2's scores, i.e. 6000 similarity scores concatenated from the 10 splits, 
together with a subset list of these challenging pairs, that were misclassified 
by our system. Each mislabeled pair presents a rather unique challenge and 
therefore we estimate the risk of overfitting from studying these pairs as rather 
low. However, it is important to evaluate performance on these pairs only for 
systems that also achieve good performance in the official LFW benchmark. 
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